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Abstract. We present some recent results on smooth vectors for unitary ir- 
reducible representations of nilpotent Lie groups. Applications to the Weyl- 
Pedersen calculus of pseudo-differential operators with symbols on the coad- 
joint orbits are also discussed. 



1. Introduction 

'Weyl-Pedersen calculus' is the name proposed in |BB09cj for the remarkable 
correspondence a t— > Op 7r (a) constructed by N.V. Pedersen in |Pe94j as a gener- 
alization of the pseudo-differential Weyl calculus on R ra . Here ir: G — > 38{Jl?) is 
any unitary irreducible representation of a connected, simply connected, nilpotent 
Lie group G, the symbol a can be any tempered distribution on the coadjoint or- 
bit 6 corresponding to 7r by the orbit method of [Ki62| . and Op"(a) is a linear 
operator in the representation space , which is in general unbounded. When 
7r is the Schrodinger representation of the (2n + l)-dimensional Heisenberg group, 
the correspondence a i— ► Op"' (a) is precisely the calculus suggested by H. Weyl in 
[We28j for applications to quantum mechanics. This calculus was later developed 
by L. Hormander in [Hor 79j and made into a powerful calculus of pseudo-differential 
operators on R"; see |Hor07j . 

In the present paper we discuss the classical notion of smooth vectors — and the 
related notion of smooth operators — with a view toward their crucial importance 
for the Weyl-Pedersen calculus. We then approach a related circle of ideas that 
recently emerged in BB09c , namely the modulation spaces for unitary irreducible 
representations of nilpotent Lie groups. We take the opportunity of this discussion 
to extend some known facts to a setting where they hold true in a natural degree 
of generality (see for instance Theorem 13.131 below). We also take a close look at 
some new examples of unitary irreducible representations and find out their related 
notions which illustrate the main theme of the present paper: the preduals of the 
corresponding coadjoint orbits, their ambiguity function, or their space of smooth 
vectors (see Proposition 15.61 and Corollary 1 5. 7j) . 
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Let us mention that the importance of the Weyl-Pedersen calculus and the re- 
lated circle of ideas goes far beyond the framework of representation theory of nilpo- 
tent Lie groups. Many other interesting developments within the theory of partial 
differential equations and the finite-dimensional Lie theory can be found for in- 
stance in th^references |^^_gn^ i _^^80], [Mi82], [Meggj . [How84j . |HRW84j . 
}HN85| . |Mi86j . [FG92] . [Man91j . |Man95j . and [Mai07] . Moreover, one can use 
a similar construction even for representations of certain infinite- dimensional Lie 
groups in order to provide a geometric explanation for the gauge covariance for the 
magnetic Weyl calculus of |MP04j . [IMP07] . [IMP09] . |MP09j and the references 
therein. The representation theoretic approach to the magnetic Weyl calculus has 
been taken up in the papers |BB09aj . |BB09bj : see also the survey |BB09dj . 

The structure of the present paper is summarized in the following table of con- 
tents: 
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Notation and background. Throughout the paper we denote by ^(Y) the 
Schwartz space on a finite-dimensional real vector space "f. That is, ^(Y) is the 
set of all smooth functions that decay faster than any polynomial together with their 
partial derivatives of arbitrary order. Its topological dual — the space of tempered 
distributions on "f — is denoted by ^'(Y). We shall also have the occasion to 
use these notions with V replaced by a coadjoint orbit of a nilpotent Lie group. 
In this situation we need the notion of polynomial structure on a manifold; see 
Sect. 1 in |Pe89| for details. We use (•, •} to denote any duality pairing between 
finite-dimensional real vector spaces whose meaning is clear from the context. 
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We shall also use the convention that the Lie groups are denoted by upper case 
Latin letters and the Lie algebras are denoted by the corresponding lower case 
Gothic letters. 

As regards the background information for the present paper, we refer to }Hor07) . 
|Fo89j . and |Gr01j for basic notions of pseudo-differential Weyl calculus on R". The 
necessary notions of representation theory for nilpotent Lie groups (in particular, 
the correspondence between the coadjoint orbits and the unitary irreducible repre- 
sentations) can be found in [Pu67] . |CG90j . and [KlOi] ; see also |Wa72j and [Ki76] . 
Our references for topological vector spaces, nuclear spaces, and related topics are 
[SchBBj . [Tr67] . and [OoM] . 

2. Smooth vectors for Lie group representations 

The smooth vectors have been a basic tool in representation theory of Lie groups; 
see for instance the early paper |Ga47j and the classical monographs |Wa72j and 
|Ki76j . In this section we discuss some of the very basic properties of the smooth 
vectors for the purpose of providing the necessary background for the later devel- 
opments in the present paper. 

Notation 2.1. Throughout this section we shall use the following notation: 

• G is a connected unimodular Lie group with the Lie algebra g; 

• dx denotes a fixed Haar measure on G; 

• "f and *3/ are some complex Banach spaces; 

• 7r: G — y Sfl^) is a representation which is continuous, in the sense that 
for every x E & the mapping tt(-)x: G — > is continuous. 

□ 

2.1. Distribution theory on Lie groups. Some references for distribution the- 
ory on Lie groups are |Br56] and |Wa72j . The present subsection just records a few 
basic notions and properties needed later. 

Definition 2.2. We define the spaces of test functions on the Lie group G as 
follows: 

(1) The space 

g(G, y) := {(f>: G -> V \ <p is smooth} 

with the usual topology of a Frechet space (given by the uniform conver- 
gence on compact sets of functions and their partial derivatives in local 
charts) . 

(2) The space 

®(G, y) := {4> E <?(G, y) I supp <f> is compact} 
with the usual topology of an inductive limit of Frechet spaces. 

If y = C then we denote simply S{G, C) = S{G) and @{G, C) = 9{G). For every 
integer m > 1 wc shall also need the function space 

^ m (G) := {4>: G — > C | <p is of class c £ m and supp <fi is compact} 

with its usual topology of an inductive limit of Banach spaces. 
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We then define the spaces of vector valued distributions 

2>'^{G, V) := {u: 3>(G, f) -> W \ u is linear and continuous} 

and 

S' 9 (G, f) := {u: S(G, f) -» <& \ u is linear and continuous} 

and endow them with the topology of pointwise convergence. We denote the eval- 
uation mapping by 

(•, •) : &*(G, f) x 9(G, f) - ST, (u, 4>) := u(4>), 
and similarly for (■, •} : S m (G, f) x d?(G, Y) -> W. 

For V = C we denote simply S m (G,C) = i m '(G) and @' W (G,C) = 9 I<& (G). 
If also W = C, then we further denote S m (<G) = S'(G) and 3>' & (G) = &(G). □ 

Definition 2.3. The support of the distribution u G £F (G, Y) is denoted by 
supp u and is defined as the intersection of all the closed sets F C G such that for 
every <f> G @(G, f) with F n supp tf> = we have (u, </>} = 0. □ 

Remark 2.4. Let L 1 (G) denote the linear space of (equivalence classes of) mea- 
surable functions on G which are absolutely integrable with respect to the Haar 
measure da; on every compact subset of G. Then there exists a natural linear 
embedding X 1 (G) @'(G). Specifically, every / G L\ oc (G) gives rise to a distri- 
bution also denoted by / and defined by 



(W>e^(G)) (/,<£) = / JWx. 

G 

Note that L\ oc (G) contains many function spaces on G, like the space of continuous 
functions, or the space of smooth functions <? (G), or the Lebesgue space L P (G) if 
1 < P < oo etc. □ 

Remark 2.5. We have 

S"(G) = {u £ @'(G) | supp-u is compact}. 

For every compact set K C G we denote S' K (G) — {u G £F'(G) | supp it C K}. □ 

Remark 2.6. We recall from |Sch66j and |Tr67j that the locally convex spaces S(G) 
and @(G) are nuclear. Moreover, we have the linear topological isomorphisms 

S(G, W) ~ £(G)®*& and 9(G, W) ~ 

which are natural in the sense that every pair (4>,y) G S'(G) x W corresponds to 
the function <fi(-)y G £(G, W). Also recall the the linear topological isomorphisms 

S(G)®S(G) ~ S(G x G) and 9(G)®9(G) ~ ^(G x G) 

that take a pair (0i,<fo) G @(G)x@(G) to the function 0i<8>02 defined by (xi, X2) 
<l>l(xi)(f>2(x2)- □ 

Example 2.7. Here are some examples of vector valued distributions that will be 
needed in the sequel. 

(1) For arbitrary g G G the ^-valued Dirac distribution 8g G S' 9 (G,^) is 
defined by 

«f:<?(G,90-jr, (%,<!>)= m- 

If ^ = C then we denote simply #;f = 5 S . 
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(2) By using Remark 12. 6[ one can define a canonical linear mapping 

£'(G) — ► S' 9 {G, W), u^u®id^, 

which takes every distribution u: <^(G) — > C to its tensor product with the 
identity operator idgr : W — > W . 

□ 

Definition 2.8. Let u\,v,2 £ S'{G). Then the tensor product of distributions 
U\®U2 £ £"(GxG) can be defined by using Remark [2~Bl such that (u\®U2, 0i<8><^>2) — 
(u\,<j)\) ■ {u2,4'2}- On the other hand, there exists a continuous linear co-product 

S'(G) — > #(G x G), <^</> A , 

where </> A (xi, £2) := 0(2:1X2) whenever x±,X2 £ G and </> £ <?(G). The convolution 
product of distributions u\ * U2 £ S"(G) is defined by 

(V0£^(G)) (ui*u 2 ,^> := (ui®u 2 ,^ A ). 

The convolution product makes the distribution space S"(G) into an associative 
algebra whose unit element is the Dirac distribution 61 £ S"(G). □ 

Example 2.9. Let us consider a few simple properties of the convolution product. 

(1) For arbitrary 31,32 £ G we have 5 gi * S g2 — S gig2 . 

(2) For every ui,u 2 £ S"(G) we have 

supp (tii * U2) C {x\X2 I Xj £ supply for j = 1, 2}. 

□ 

Definition 2.10. We shall think of g as a real subalgebra of its complexification 
gc '■= C CS)r 9, hence flc = + ifl- The universal enveloping algebra U(gc) is the 
complex unital associative algebra satisfying the following conditions: 

(1) The complexification gc is a Lie subalgebra of U(fjc)- 

(2) For every complex unital associative algebra srf and every linear mapping 
8:gc->*f satisfying 6([X,Y]) = 9{X)9{Y) - 9{Y)6{X) for all X,Y e Qc 
there exists a unique extension of 9 to a homomorphism of complex unital 
associative algebras U(gc) —* ^ ' ■ 

One can prove that there always exists an algebra U(flc) satisfying these conditions 
and it is uniquely determined up to an isomorphism of complex unital associative al- 
gebras. Moreover, there exists a unique (complex-)linear mapping U({jc) ~ * U(gc)j 
u t-^ u 1 - such that 

(vw) 1 - = w^v^, (w^-) 1 - = w, and X 1 - = -X 

for every u, v £ U(gc) an d X £ g. (See [Dix74j for more details). □ 

Example 2.11. If g is an abelian Lie algebra of dimension n, then U(gc) is 
the algebra of polynomials C[xi, . . . , x n ] and for every p £ C[xi, . . . , x n ] we have 
p x (xi, . . . , x n ) = p(-xi, . . . , -x n ). □ 

We are going to describe in Remark 12.131 some realizations of the universal en- 
veloping algebra U(jjc) which are needed later. To this end we first introduce the 
regular representations of G on distribution spaces. 
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Definition 2.12. Wc shall need the translation x i— ► gx and 

p g : G —> G, x xg defined by an arbitrary element g 6 G. The corresponding 
regular representations of G on the distribution space @'(G) are defined by 

A : G -> End (^'(G)), (A(p)it, 0) = {u,<po X g ) 

and 

p: G -> End (^'(G)), (p( 5 )u, 0) = (u,cf>o p g -,) 
whenever g £ G, u £ S#'{G), and cj> £ ^(G). For every leg and £ £{G) we 
also define the functions 



\{X)4>:G->V, (X(X)d>)(z) = A ( , (( . M , ( .(-/.V):) 



=o 



and 



p(X)0: G -» C, = 1 cj>(zexp G (tX)). 

at t=o 

Then we can define the derivatives of the regular representations by 
A: 0-»End(0'(G)), (\(X)u,(j>) := (u,\(-X)<P) 

and 

p: g - End (S>'(G)), (p(X)u, <j>) := (u, 
These derivatives are homomorphisms of Lie algebras, hence condition @ in Defi- 
nition 12.101 shows that they can be uniquely extended to unital homomorphisms of 
associative algebras U(jjc) — ¥ End(£F'(G)). These extensions will also be denoted 
by A: U(0 C ) -> End(^'(G)) and p: U(fl C ) -> End(^'(G)), respectively. 

For later use, we also introduce the notation <j>^{x) := for every <j> £ #(G) 

and x £ G. This gives rise to the idempotent linear mapping 

@'{G)^@'{G), un-ir 1 , 

where (it 1 , (j)) := (it, ^) for u £ 0'(G) and £ 9{G). □ 

Remark 2.13. With Definition 1 2 . 1 21 at hand, we can describe some realizations of 
the universal enveloping algebra U(gc) as follows. For the sake of simplicity, let us 
denote by $[{G) the space of distributions on G with the support contained in {1}, 
thought of as a complex unital associative algebra with respect to the convolution 
product, cf. Example l2.91 (This set should actually be denoted by S'^iG) according 
to Remark 12.51 ) Recall that <5i £ S"(G) is the Dirac distribution at 1 £ G. 

Both mappings 

U(flc)^<(G), w^X(w)5 x , 

U(flc)-»W), w^p(w)Si 
are isomorphisms of complex unital associative algebras. (See for instance Th. 1 in 
Sect. 10.4 of |Ki76j .) These isomorphisms are related by the commutative diagram 

U( 0C )^U( 0C ) 

A(-)5i p(-)<5i 

^(G)^t^(G) 



where the horizontal arrows stand for the mappings introduced in Definitions 12.101 
and 12.121 respectively. From now on, we perform the identification U(jjc) — &\{G) 
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by means of the mapping w p(w)S±, by writing simply w instead of p(w)Si 
whenever w G U(gc)- D 

Proposition 2.14. For every integer m > 1 and every compact neighbourhood K 
of 1 G G there exist finitely many elements Ui, . . . ,un G U(gc) an d the functions 

N N 

4>i, . . . , (f>N G ^"(G) |J supp^j C K such that 5\ = J] 3 * Uj. 

i=i " j'=i 

Proof. Use Lemme 2 in |Ro76] or Lemma 2.3 in [DDJP09]; see also the proof of 
Lemme 1.1 in |Ca76j . □ 

2.2. Smooth vectors. 

Definition 2.15. The smooth vectors for the representation it: G — > 3§(W) are 
the elements of the linear subspace of W defined by 

W oc -.= { y eW\Ti(-)y^S{G,W)}. 

The linear space Wx> will be endowed with the linear topology which makes the 
linear injective map 

Wx, — > &(G, W), y~ir(-)y 
into a linear topological isomorphism onto its image. 

For every distribution u G S"(G) and every smooth vector y G Wx we then define 

■k(u)y :— (u®idgr,Tr(-)y) G W . 

□ 

Proposition 2.16. The following assertions hold: 

(1) The space of smooth vectors Woo is a Frechet space and the inclusion map 
Woo c — ► W is continuous. 

(2) The space Woo is dense in W. 

(3) For every u G £'(G) we have ■k{u)&o C g^,. 

(4) TTie mapping ■k: §' '(G) — > End(^3o) is a homomorphism of unital associa- 
tive algebras. 

(5) for every l£g and y G we /iawe n(X)y := -4 7r(exp G (tX))y. 

{=0 

(6) For every y G we have a smooth mapping 7r(-)y: G — > 

Proof. See for instance |Wa72j and Sect. 10.5 in jKi76] . □ 

Notation 2.17. We shall always denote by W-oo the space of continuous antilincar 
functionals on the Frechet space Woo- O 

Proposition 2.18. For every integer m > 1 i/iere exisi finitely many functions 
4>i, . . . , 4>n G ^"(G) such that for every y G Woo there exist j/i, . . . , yjv G ^ saizs- 
/ymg i/ie equality y = -k{(j>i)yi H h ir(<j) N )y N . 
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Proof. Use Proposition 12.141 to get ui,...,un G U(flc) and <f>i, . . . ,(j>N 6 ^™(G) 

N 

with S± = ^2 fij * u j ■ Then Proposition 12.161 shows that 

N N 

where we have denoted j/j :— 7t(uj)y for j = 1, . . . , N. □ 

Remark 2.19. As we already mentioned, the smooth vectors for representations 
of Lie groups were discussed in detail in |Wa72| . Other important references in 
this connection are [Ga47] , [Ga60] . [Ki76j . [Ca76] . [CGP77j . [UM78j . [RT87] . and 
[CG90] , □ 



3. Smooth operators for unitary representations 

We are going to discuss here the space of smooth operators for a given representa- 
tion of a Lie group. The method of investigation was suggested in [Pc94j and relies 
on exhibiting this space of operators as the space of smooth vectors for a suitable 
representation (see Definition I3.11[) . The main result is recorded as Theorem 13.131 
and it is particularly significant in the case of unitary irreducible representations 
of nilpotent Lie groups fCorollary |3.14p . 

Notation 3.1. In this section we shall use the following notation: 

• G is a connected unimodular Lie group with the Lie algebra g; 

• dx denotes a fixed Haar measure on G; 

• J%? is a complex Hilbert space; 

• 7r : G — y SS{J^f) is a continuous unitary representation, and is the 
corresponding space of smooth vectors. 

□ 

The following notion of smooth operator was singled out on page 349 in |How77j 
and then further developed in |Pe94j . 

Definition 3.2. The set 3§{J#') 00 of smooth operators for the representation 7r is 
defined as the set of all operators T £ 3§{Jff) satisfying the following conditions: 

(1) We have T(Jt?) + T*(J?) C J^. 

(2) For every u € U(jjc) the operators ir(u)T and n(u)T* are bounded on Jjf. 

□ 

Example 3.3. For every 1,56 the rank-one operator (• | x)y belongs to the 
space of smooth operators S^(M' S ) 00 . We shall see in Corollary 13.161 that the linear 
span of these rank-one operators is dense in ^(J^') 00 provided that G is a nilpotent 
Lie group and 7r is an irreducible representation. □ 

Remark 3.4. It follows at once by Definition 13.21 that £i§(,3%') 00 is an associative 
*-subalgebra of 9B{ ) . □ 
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Definition 3.5. We shall say that the representation 7r has a smooth character if 
for every tj> £ 9(G) we have tv(4>) s &i(J4?) and the linear mapping 

9(G) -> 6i(je), 0^tt(0) 

is continuous. In this case we define the corresponding character as 

X,:9(G)^C, X *(4>) ■= Tr 

Note that Xtt € 9'(G). □ 

Example 3.6. Every unitary irreducible representation of a nilpotent Lie group 
has a smooth character; see for instance Th. 2 in §5 of Ch. II, Part. II in |Pu67j . □ 

Remark 3.7. If the representation 7r has a smooth character, then there exists a 
continuous seminorm p(-) on 9(G) such that 

(V<f>e9(G)) |[7r(0)||i <p(0). 

In view of the definition of the topology on 9(G) and of the fact that 9(G) is dense 
in ^ m (G) for every m > 1, it then easily follows that for every compact subset 
K C G there exists an integer m > 1 such that for every G ^"(G) fl S' K (G) we 
have tt(0) £ &i(J4f), and moreover the mapping 

<*f m (G) n £' K (G) -» 6i i — ^ jt(0) 

is linear and continuous. □ 

Definition 3.8. An admissible ideal is a non-trivial two-sided ideal ^ of 33(34?) 
with the following properties: 

(1) The ideal fl is endowed with a complete norm || • \j such that for every 
A,Be SS(.ye) and T e ^ we have H^TEH^ < ||A|| • ||T||^ ■ ||S|| and 
\\T*\\ f = \\T\\ f . 

(2) The ideal ^(M*) of finite-rank operators is a dense subspace of fl . 

(3) For every i,f/gif we have ||(- | x)y\\^ = \\x\\ ■ \\y\\. 

□ 

Example 3.9. Every Schatten ideal & P (J$?) with 1 < p < oo is an admissible ideal. 
There exist many other examples of admissible ideals; see for instance |GK69j . □ 

Remark 3.10. Let fl be an admissible ideal. By using condition @ in Defini- 
tion [3TS] with A = idjf? and B = (■ | x)x for x G JV, and then taking into account 
condition J3]), it follows that ||Tx|| < \\T\\j ■ \\x\\. That is, for every T e fl we 
have ||T|| < \\T\\j. 

On the other hand, it follows at once by condition in Definition 13.81 that for 
every T 6 J?(JT) we have ||T||^ < ||T||i. Since ^(Jf) is dense in 6ipf ), we get 

(vresi(jn) imi < < imii. 

In particular, we have &i(J$? ) Q fl ■ □ 

Definition 3.11. For every admissible ideal fl we define a linear representation 
tt® 2 : G xG^J(/) by 

7>"5?(ffi,S2)T := 7r( 5 i)T7r( 52 ) _1 
for every 51,32 6 G and T € fl . □ 
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Lemma 3.12. The representation tt^, : G x G — > ^( c f) is continuous for every 
admissible ideal J! . 

Proof. If T G ^(3t), then it is straightforward to check that tt®?(-)T: GxG ^ J 
is a continuous mapping. 

Now let T G J! arbitrary. Since ^ is admissible, there exists a sequence {Tk}k>i 
in ^(J^) such that lim \\T — Tk\\<? = 0. On the other hand, since tt is a unitary 

representation, it follows that for k = 1, 2, . . . and every (91,92) 6GxGwe have 
lk5?(5i,52)T-4 2 ( 5l , 32 )r fc ||^ < ||T-T fe ||^. 

Therefore tt® 2 (-)T: GxG^ ^ is the uniform limit on G x G of the sequence of 
continuous mappings 7r^r 2 (-)T, hence it is in turn continuous. □ 

Theorem 3.13. Let 7r: G — > ^(J^) &e a continuous unitary representation, as- 
sume that J? C 33{3l?) is an admissible ideal, and denote by the space of 
smooth vectors for the corresponding representation 7r^? . Then the following asser- 
tions hold: 

(1) We have Jf x C S8(3f)^. 

(2) // the Frechet space of smooth vectors is nuclear, then we have 

and the Frechet space J>oo does not depend on the choice of the admissible 
ideal J! . 

Proof. ([]]) To prove the inclusion C 38(J%?) 00 , let T G Jfi^ arbitrary, hence 
the mapping 

GxG^ J, ( gi ,g 2 ) ~ n® 2 ( gi ,g 2 )T = 7r(g 1 )Tn(g 2 )- 1 

is smooth. In particular, the mapping tt(-)T: G — > ^ is smooth. On the other 
hand, it follows by Remark 13.101 that for arbitrary x G Jif we have a continuous 
linear mapping ^ — > Jf, T i— > Tx. Hence the mapping tt(-)Tx: G — > Jif will be 
smooth as a composition of two smooth mappings. Thus for arbitrary x G Jif we 
have Tx € Jfoo- Moreover, since the operation of taking the Hubert space adjoint 
is (M- linear and) continuous on by condition (TT|) in Definition 13.81 it follows at 
once that T* G ^foo . Hence by the above reasoning with T replaced by T* we get 
T*x G M'oo for arbitrary x G 3% '. Thus the operator T satisfies condition |T|) in 
Definition 13.21 To check condition {2} in the same definition just note that since 
the mapping 7r(-)T: G — ► Ji is smooth, it follows that for every u G U(gc) we have 
7t(w)T G ^ hence n(u)T G 38 (J?) . Since we have seen above that T* G ,/oc, 
it also follows that -k(u)T* G 38 (3f). This completes the proof of the fact that 
T G 38(3?)^. 

(0 If is a nuclear space, then the inclusion map is a nuclear 

operator (see Prop. 7.2 in Ch. Ill of Sch66 ). Since condition |T]) in Definition 13.21 

shows an arbitrary operator T G ^(J^)^ factorizes as 3¥ — ► 3^, ^ , it follows 
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that T e 6i(Jf) (see Cor. 2 to Prop. 7.2 in Ch. Ill of |Sch66| ). Thus, by taking 
into account the above Assertion |T|), we get 

Joo C 3§(Mf) ao C 6i(JT). (3.1) 

To see that J?oo does not depend on the choice of the admissible ideal J 1 , we shall 
prove the equality of Frechet spaces 

Jfoo = 6i(Jf)oo, (3.2) 

where the right-hand side denotes the space of smooth vectors for the representation 
w e>i(je) '■ G x G ^ 38(<&i(M?)). First recall from Remark [3T0l that we have a con- 
tinuous inclusion map &i(Jt?) c — > ^ , which clearly intertwines the representations 
7T 6i(M') an< ^ 7r 5 f2 ' ^ then easily follows by Definition 12 . 1 51 that we have a continuous 
inclusion map for the corresponding spaces of smooth vectors &i(34?) 00 <—* J?oo- 
On the other hand, we have already proved that Jioo C &i(Jf), hence there exists 
the following commutative diagram 



whose arrows stand for inclusion maps between Frechet or Banach spaces. The 
arrows that point to J> are continuous inclusions (by Remark 13.101 and Propo- 
sition [2TT6)JT|) ) , hence the closed graph theorem implies that the inclusion map 
J^oo c — > &i(J$?) is continuous as well. Since for arbitrary T £ the mapping 
G x G — > ^foo, (51,52) > 7r (5i)7 1 7r(g 2 ) _1 is smooth by Proposition I2.16t [fj|). it 
then follows that the mapping G x G — » 6i(if), (91,92) l_ ► 7r (5i)7 1 7r(g2)~ 1 is also 
smooth, hence T G 6 1(^)00. Thus = ©i(J£*)oo as sets. Since both sides of 
this equality are Frechet spaces and we have already seen that the inclusion map 
&i(3# y ) QO ^foc is continuous, it follows by the open mapping theorem that we 
have the equality of Frechet spaces in 



Finally, note that for arbitrary T G 3^(Jf)oo and every u G U(gc) we have 
■k(u)T, -k(u)T* £ 3$(M') QX > (see Definition 13. 2p . On the other hand, we have proved 
above that 38(3?)^ C &x(Jf), hence %(u)T, n(u)T* G <&i(M?) for all u G U(flc), 
and this implies that T G & 1(^)00- Thus 38(3?)^ C 61(^)00, and then by 
using Assertion Q with = 61 (Jf) we get 38(3^)^ = 61 (JT)^. Now by (gj) 
and (|3.1[) we get = S^[M >S ) 00 C &\(Jf?), and this completes the proof. □ 

Corollary 3.14. Assume £/iai G is a nilpotent Lie group and tt: G — ► 38(3?) is a 
unitary irreducible representation. If ^ C 38(3?) is an admissible ideal, and we 
denote by the space of smooth vectors for the corresponding representation "k®j , 
Then the following assertions hold: 

(1) The Frechet space of smooth vectors is nuclear. 

(2) We have = 33(34?) ^ C &x(3?), and the Frechet space ^foo does not 
depend on the choice of the admissible ideal J" . 

(3) The space of smooth operators 38(3?) 00 has the natural structure of a nu- 
clear Frechet space. 
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Proof. Since the representation tt is irreducible, there exists a linear topological 
isomorphism from the Frechet space ■M'oa onto the Schwartz space of rapidly de- 
creasing functions y(R d / 2 ), where d is equal to the dimension of the coadjoint 
orbit of G corresponding to the representation tt. (This follows by Th. 1 in §5 of 
Ch. II, Part. II in [Pu67] : see also the Cor. to Th. 3.1 in [CGP77j . or |CG90j .) On 
the other hand, it is well known that the Schwartz space ^(R d ^ 2 ) is nuclear (see 
for instance Ex. 5 in §8 of Ch. Ill in [Sch66 ). Therefore the Frechet space Jffoo is 
nuclear, and then Theorem 13. 131 applies. 

Finally, by using Assertion when J* — (the Hilbert-Schmidt ideal), 

it follows that ^(^)oo is equal to the space of smooth vectors for the unitary 
representation tTq^^)' nence it is a Frechet space in a natural way. On the other 
hand, the representation fg^^) ^ s irreducible since so is tt. (See for instance 
the proof of Lemma 2.18(a) in BB09cJ.) Now the fact that S§(,3 I ^') 00 is nuclear 
follows by the above Assertion ([I]) applied for the unitary irreducible representa- 
tion Trf^jpj : G x G -> 38{& 2 {.JI?)). □ 

Corollary 3.15. If G is a nilpotent Lie group and tt: G — > is a unitary 

irreducible representation, then the operators in ^(J^') 00 are precisely the regular- 
izing operators. That is. A 6 £i§(Jtf ) 0o if and only if A extends to a continuous 
linear map A: Jff-oc — > Jt?oo, so that the diagram 



is commutative 

Proof. By the closed graph theorem, it is enough to prove that if A £ B8{M') oa 
and / € Jtf'-ooi then Af 6 Jrffoc, hi the sense that there exists a smooth vector 
denoted Af such that for every E Jf?oo we have (/ | A*(f>) = (Af | <p). This 
is a consequence of the above Proposition I2.16lj3j ). Corollar ^3.14[ and Th. 1.3(b) 
in [Ca76] . 

Conversely, it follows by Definition 13.21 that the restriction to Jt? of every con- 
tinuous linear map A: JtfLoo — > J^oo belongs to ^(,yf)oo. □ 

Corollary 3.16. Assume that G is a nilpotent Lie group and it: G — > £$(Jf?) 
is a unitary irreducible representation. The linear space spanned by the operators 
(• | x)y with x,y 6 Jtfoo is dense in ^(Jtf)^. 

Proof. Let T e @(M') 00 arbitrary. Then Corollary l3~14l|2"j) shows that T is a 
smooth vector for the representation tTq 2 /^ : G x G — * ^(62 G^))- It follows by 
Proposition 12. 181 that there exist finitely many functions (f>i, . . . , 4>n £ ^^(G x G) 
and the operators Y\, . . . , Yn G 62 {^) such that 

Since @(G x G) is dense in ^™(G x G) and 2>(G) ® S>{G) is dense in 9(G x G), 
it follows by Proposition 12 . 161( 1)) that T can be approximated in 38(34? ) oa by finite 
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linear combinations of operators of the form 

with ipi,tp2 S @(G) and Y e &2(J^')- On the other hand, such an Y can be 
approximated in &2{J^') by finite linear combinations of operators (■ | V2)v\ with 
V\,V2 £ Jf?. The corollary now follows by noticing that 

tt(-0i)((- I v 2 )vi)iT('ip 2 L ) = (• | 7r(V^-)v 2 MV>i)vi 
and recalling that n(^)v 6 when V 6 ^(G) and ue Jf ( |Ga47| ). □ 

Remark 3.17. Let J! C BS{3%') be any admissible ideal. If the representation 
7r has a smooth character, then the corresponding space of smooth vectors is 
nuclear according to Th. 2.6 in [Ca76], hence the above Theorem 13. 13[j 2l applies. 

The inclusion C &i(Jf?) can be alternatively proved in this case as follows. 
Let T 6 ^foo arbitrary. Since the representation 7r has a smooth character, we have 
a continuous linear mapping 

3>{G) -»6i(JT), tt(0). 

On the other hand, note that for every <j>\,<j>2 6 ^(G) we have 7r^ 2 (</>i <g> <j>2)T — 

7r(0i)T7r(<^2 )• Hence for arbitrary F g / we get a jointly continuous trilinear 
mapping 

0(G) x 0(G) x J -> 6i(JT), (</>i,<h,Y) ^^ 2 (0i <8> &)Y, 

which extends to a jointly continuous bilinear mapping 

®(GxG)x/^6!(Jf), (0, Y) i * ^ 2 (0)Y". 

(See also Remark |2. 61 ) By an argument similar to the one of Remark l3.7i we can 
further extend the above mapping to a continuous bilinear mapping 

(<^ m (G x G) n S' K {G x G)) x J -> Si(JT), (0, Y) h-> 7r® 2 (0)Y (3.3) 

for a suitable compact neighborhood if of (1,1) 6 G x G and a suitably large 
integer m > 1. On the other hand, since T £ ^oo, it follows by Proposition 12. 181 
that there exist finitely many functions <f>\, . . . , 4>N S ^"(G x G) (~l $' K (G x G) and 

the operators Y x , . . . , Yv G ^ such that T = ny((f) 1 )Y 1 H h 7r^f(0jv)ijv, hence 

by (O we get T e Si(Jf). □ 



4. Weyl-Pedersen calculus 

In the present section we provide a brief discussion of the remarkable Weyl cor- 
respondence constructed in [Pe94] and we shall also describe some complementary 
results which were recently obtained in [BB09c . 

4.1. Preduals for coadjoint orbits. This subsection records some properties 
of the coadjoint orbits of nilpotent Lie groups which play a crucial role for the 
construction of the Weyl-Pedersen calculus. 

Setting 4.1. We shall use the following notation: 
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(1) Let G be a connected, simply connected, nilpotent Lie group with Lie 
algebra g. Then the exponential map exp G : g — > G is a diffcomorphism 
with the inverse denoted by log G : G — > g. 

(2) We denote by g* the linear dual space to g and by (•,•}: 0* x g — > M the 
natural duality pairing. 

(3) Let £o G g* with the corresponding coadjoint orbit 6 :— Ad G (G)£o Q g*. 

(4) Let 7r: G — > BS{J^) be any unitary irreducible representations associated 
with the coadjoint orbit by Kirillov's theorem ( |Ki62j ). 

(5) The isotropy group at £o is G^ :— {g £ G \ Ad G (g)£o = £o} with the 
corresponding isotropy Lie algebra g^ = {X G g | £o ° ad g X = 0}. If we 
denote the center of g by 3 := {X G g | [X, g] = {0}}, then it is clear that 

(6) Let n := dimg and fix a sequence of ideals in g, 

{0} = go C fli C • • • C Q n = g 

such that dim(gj/0j_i) = 1 and [g, Qj] C for j = 1, . . . , n. 

(7) Pick any Xj G Qj \ Qj-% for j = 1, . . . , n, so that the set {Xt, . . . , X n } will 
be a Jordan- Holder basis in 0. 

□ 

Definition 4.2. Consider the set of jump indices of the coadjoint orbit & with 
respect to the aforementioned Jordan-Holder basis {Xi, . . . ,X n } C 0, 

e := {j 6 {l,...,n} I 3 - g 0j_i + g 5o } = {j G {1, . . . , n} | X; £ 0J _i + g s J 

and then define the corresponding predual of the coadjoint orbit 6 

e := span{X, | j G e} C g. 

We note the direct sum decomposition g = 0{ o + e - □ 

Remark 4.3. Let {£1, . . . , £„} C 0* be the dual basis for {Xi, . . . , X n } C 0. Then 
the coadjoint orbit 6 can be described in terms of the jump indices mentioned in 
Definition 14.21 More specifically, if we denote 

tie ■= span{£j j G e} and g@ := span{^ | j <£ e}, 

then the coadjoint orbit C g* ~ g* x gjj- is the graph of a certain polynomial 
mapping g* — > g^. This leads to the following pieces of information on 

(1) dim if? = dimg e = carde =: d; 

(2) if we let j% < ■ ■ ■ < jd such that e — {ji, . . . ,jd}, then the mapping 

is a global chart which takes the Liouville measure of & to a Lebesgue 
measure on M d . 

We define the Fourier transform 5^(0) — ► =5^(g e ) by 

(V* G g e ) a(X) = y e-'^'^aCOd^ 
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for every a 6 S^iff), where d£ stands for a Liouville measure on G ' . This Fourier 
transform is invertible. The Lebesgue measure on g e can be normalized such that 
the Fourier transform extends to a unitary operator 

L\ff) _> L 2 ( Se ), a^a, 

and its inverse is defined by the usual formula. We shall always consider the predual 
Q e endowed with this normalized measure. (See for instance Lemma 1.6.1 in |Pe89] 
and Lemma 4.1.1 in |Pe94] for more details and proofs for the above assertions.) □ 

Remark 4.4. Some basic references for the geometry of coadjoint orbits of nilpo- 
tent Lie groups include [Pu67] . [Pe84] . [Pe88] . [Pe89] . and |CG90j : see also [BB09dj . 

□ 

4.2. Weyl-Pedersen calculus and Moyal identities. We begin this subsection 
by the general construction of a Weyl correspondence due to |Pe94] . 

Definition 4.5. The Weyl-Pedersen calculus Op"(-) for the unitary representa- 
tion 7T is defined for every a G ,5^(0) by 

Op 77 {a) = J a(X)ir(exp G X)dX G &(Jf?). 

Br 

We call Op* (a) is the pseudo-differential operator with the symbol a G □ 
Theorem 4.6. The Weyl-Pedersen calculus has the following properties: 

(1) For every symbol a G we have Op" (a) G 38(3% : ') 00 and the mapping 

— > ^(^)oo ; a i— > Op" (a) is a linear topological isomorphism. 

(2) For every T G &(J4?) 00 we have T = Op"(a), where a G satisfies 
the condition a(X) = Tr (7r(exp G X) _1 T) for every X G e - 

(3) For every a, b G 5^(0) we have 

(a) Op" (a) = Op" (a)*; 

(b) Tr(Op"(a))=/a(£R; 

(c) Tr(Op"(a)Op"(6)) =/a(0« 

<? 

(d) Tr (Op" (a) Op" (&)») = J a(0&(0<tf- 

Proof. See Th. 4.1.4 and Th. 2.2.7 in [Pe94] , □ 

Definition 4.7. Recall from Remark 13.41 that is an involutive associa- 

tive subalgebra of 31(31?) . It then follows by Theorem I4.6t fl~j) that there exists an 
uniquely defined bilinear associative Moyal product 

y(^)xy(^)^y(^, (a,6)^a#"o 

such that 

(Va, b G Op"(a#"o) = Op"(a)Op"(o). 

Thus y(0) is made into an involutive associative algebra such that the mapping 
y(&) £%(J%?)oo, a i— ► Op" (a) is an algebra isomorphism. □ 
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Notation 4.8. Recall that J^oo is the space of continuous antilinear functionals 
on M'oa and the corresponding pairing will be denoted by (• | •) : ffl-oo x J^x, — » C. 
just as the scalar product in iff , since they agree on J^x, x M'qq if we think of 
the natural inclusions Jtifoo °-> <— > J^_oo- (See for instance |Ca76| for more 
details.) □ 

Definition 4.9. If / G M'-cx, and £ or /, 6 then we define the 

corresponding ambiguity function 

>) = e - C, = (/ | 7r(exp G X)0). 



For G J^-oo and / G we also define (.s/ ( j,f)(X) — (</> | 7r(exp G (— X))f) 

whenever X £ g e . 

It follows by Proposition OnKD below that if /, </> G then ^/ G L 2 (fl e ), 
so we can use the aforementioned Fourier transform to define the corresponding 
cross-Wigner distribution W(f, <j>) G L 2 {G) such that W{f^4>) := ^f. □ 

The second equality in Proposition l4.101f Tj) below could be referred to as the 
Moyal identity since that classical identity (see for instance |Gr01j ) is recovered in 
the special case when G is a simply connected Heisenberg group. 

Proposition 4.10. The following assertions hold: 

(1) IfcfreJif, then xfyf G L 2 (q c ). We have 

hh I ^4>2h)L 2 (g c ) = (/i I h).^ ■ {fa | fa)je 



= (W(h,fa)\w(f 2 ,fa)) LH&) (4 ' 1} 

for arbitrary fa, fa, fi, fi G JM? . 
(2) If(f> G J$? with |j0o|| = lj then the operator stf$ : — ► L 2 (g e ), f t— > £^<p f, 
is an isometry and we have 



.J)(X)-Tr(exp G X)falX = (<j>\fa)f 

/or every (f> G / € •-^ £ '- ^Ti particular, 

(^J)(X)-7r(e X p G X)fadX = f 



for arbitrary f G 

Proof. See |BB09cj . □ 
Corollary 4.11. T/ie following assertions hold: 

(1) For eac/i a G we have 

(Op»0 | f)jf = (a | rftfi&b.) = (" I ^(/.0))^(^) 
whenever <fi, f G Similar equalities hold if a £ ,9" (ff) and <fi, f G J£^o ■ 

(2) If fa, fa G and a := W(fa,fa) G S fi {&), then Op*" (a) is a rank-one 
operator, namely Op 7r (a) = (• | fa) fa. 

Proof. See |BB09cj . □ 
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Assertion <j3j> in the following corollary in the special case of square-integrable 
representations reduces to a theorem of |Co84j and |CM96j . One thus recovers 
Th. 2.3 in [GZOl] in the case of the Schrodinger representation of the Heisenberg 
group. 

Corollary 4.12. If 4>o € .s^oo with \\4>o\\ — lj then the following assertions hold: 
(1) For every f G Jt?-oo we have 

h f)(X)-n(exp G X)<t> dX = f (4.2) 



where the integral is convergent in the weak* -topology of J^L^. 

(2) If f G J^oo, then the above integral converges in the Frechet topology of 

(3) If f G J^f-oo, then we have f G J$?oo if and only if £^<p f G y(g e )- 

Proof. See [BB09c]. □ 

Remark 4.13. Let ^(Jff)*^ be the topological dual of the Frechet space 3§{M') 00 
and denote by (•, •) either of the duality pairings 

^(.Jf)*^ x ^pr)oo -» C and x y(0) -» C. 

Then for every tempered distribution a G ,9"(&) we can use Theorem l4.6lf T|) to 
define Op 7 " (a) G ^{M')* 00 such that 

(V6 G y{0)) (Op^a),Op 7r (6)> = (a,6) 

Just as in Definition 14.51 we call Op 7r (a) the pseudo-differential operator with the 
symbol a G .9" ((f). Note that if actually a G y(ff), then the present notation 
agrees with Definition 14.51 because of Theorem I4.61|3"c|) . 

The continuity properties of the above pseudo-differential operators can be inves- 
tigated by using modulation spaces of symbols; see |BB0 9c for details. Specifically, 
one can introduce modulation spaces M^ s (tt) for every unitary irreducible repre- 
sentation rr: G -> @(Jf?). We always have C MJ s (tt) and M|' 2 (7r) = Jf. 
There exists a natural representation 7r# :G«G^ ,9§(L 2 (ff)) such that for suit- 
able $ G y(ff) \ {0}, the Weyl calculus Op 7 ^-) defines a continuous linear mapping 
from the modulation space M™ a {tt# ) into the space of bounded linear operators 
on J4?. One of the main theorems of [GH99] is recovered in the special case when 
7T is the Schrodinger representation of the (2n + l)-dimensional Heisenberg group. 
Some new results related to this circle of ideas will be established in Section [5] 
below. □ 

Remark 4.14 (|BB09c ). We can define the cross- Wigner distribution W(f\, fa) G 
y'(ff) for arbitrary /i, fa G ffi-oa as follows. By using Corollarv l3 . 1 51 we can define 
for /i,/2 G M'-ao the continuous antilinear functional 

T fuf2 : ^(JfU - C, T fltf2 (A) := (fa | Af 2 ). 

That is, T h . h G ^(Jf)*^, and then Th. 4.1.4(5) in [Pe94] shows that there exists 
a unique distribution af 1 j 2 G y'(&) such that Op 7r (a/ 1 j 2 ) = Tf 1 j 2 . Now define 

^(fij2):=a fuf2 . 
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We can consider the rank-one operator Sf lt f 2 := (• | f 2 )fi- jff'oa — * i^-oo and for 
arbitrary A £ 3§{J'iC) 00 thought of as a continuous linear map A: Jtif-oo — > ffioa as 
above we have 

Tr (S fuh A) = (f 1 \Af 2 ) = T fuh (A). 

Thus the trace duality pairing allows us to identify the functional Tf t j 2 £ ^(J^ )J C 
with the rank-one operator (• | /2)/i, and then we can write 

(V/i,/ 2 e^-oo) P 7r (^(/ 1 ,/ 2 )) = (-|/ 2 )/ 1 . (4.3) 

In particular, it follows that the above extension of the cross- Wigner distribution 
to a mapping W (■,■): Jf-oo x Jf-oe — > ,9"{@) allows us to generalize the assertion 
of Corollary [4~TTP|) to arbitrary 0i, <f> 2 £ MLoo- □ 

5. Modulation spaces 

The modulation spaces play a central role in the timc-frcqucncy analysis (see 
[GrOlj ) and proved to be a very useful tool in the study of continuity properties 
of pseudo-differential operators ( GH99]). These classical ideas can be formulated 
within the representation theory of the Heisenberg groups, and this representation 
theoretic viewpoint turned out to be very effective in order to extend the corre- 
sponding notions to unitary irreducible representations of arbitrary nilpotent Lie 
groups (sec [BB09c ) . In the first two subsections of the present section we shall pro- 
vide some preparations and then describe the general notion of modulation spaces 
introduced in [BB09c|. We eventually illustrate this notion by discussing a specific 
class of irreducible representations on Hubert spaces of the form L 2 (ff), where & is 
any coadjoint orbit of a nilpotent Lie group (see Proposition 15 . 61 and Remark l5.8p . 

5.1. Semidirect products. 

Definition 5.1. Let G\ and G 2 be connected Lie groups and assume that we have 
a continuous group homomorphism a: G\ — > AutG 2 , g\ i— > a gi . The correspond- 
ing semidirect product of Lie groups G\ x Q G 2 is the connected Lie group whose 
underlying manifold is the Cartesian product Gi x G 2 and whose group operation 
is given by 

(51,32) • (hi,h 2 ) = (gihi,6 h -i(g 2 )h 2 ) (5.1) 

whenever gj, hj G Gj for j = 1,2. 

Let us denote by a : Qi — > Der g 2 the homomorphism of Lie algebras defined as 
the differential of the Lie group homomorphism G± — > Autg 2 , g\ >— > L(a 9l ). Then 
the semidirect product of Lie algebras $i Xq Q2 is the Lie algebra whose underlying 
linear space is the Cartesian product Q\ x q 2 with the Lie bracket given by 

[{X U X 2 ), (Y U Y 2 )] = ([X u Yl], d(Xi)K a - a{Y l )X 2 + [X 2 ,Y 2 ]) (5.2) 

if Xj, Yj £ Qj for j = 1,2. One can prove that gi x„ g 2 is the Lie algebra of the Lie 
group Gi x Q G 2 (see for instance Ch. 9 in |Ho65| ). □ 

Remark 5.2. Assume that Gi and G 2 are nilpotent Lie groups and a: G\ — > 
Aut G 2 is a unipotent automorphism. That is, for every X± £ q± there exists an 
integer m > 1 such that a(Xi) m = 0. Then an inspection of (|5.2j) shows that 
0i «a fl 2 is a nilpotent Lie algebra, hence G\ x Q G 2 is a nilpotent Lie group. □ 
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Example 5.3. Let G be a nilpotent Lie group. If we specialize Definition 15.11 for 
G\ := G, G2 = (0,+), and a := AcIg : G — > Autfl, then we get the semidirect 
product G KAd G g which is a nilpotent Lie group by Remark 15.21 and is isomorphic 
to the tangent group TG. The Lie algebra of G KAd G is g x a d B 0o (where go stands 
for the abelian Lie algebra that has the same underlying linear space as g) and the 
corresponding exponential map is given by 

1 

exp GKAdGfl (A,y) = (exp G A,y Ad G {exp G (sX))Y ds) 



1 

= (exp G X,J e sad ° x Yds) 


for every (X,Y) £ g K a d B go (see for instance Prop. 2.7(2) in [BB09a| ). □ 



5.2. Modulation spaces for unitary representations. In this short subsection 
we just recall the definition of the modulation spaces for the unitary irreducible 
representations of nilpotent Lie groups. We refer to [B B09cj for a more detailed 
discussion of this notion. 

Definition 5.4. Let 4> £ J^oo \ {0} be fixed and assume that we have a direct sum 
decomposition g e = g\ + g 2 . 

Then let 1 < r, s < 00 and for arbitrary / 6 -^f-oc define 



\(^f)(x 1 ,x 2 )\ r dx 1 ) s/r dx 2 Y /s e [o,oo] 



si si 

with the usual conventions if r or s is infinite. Then we call the space 

M r /(7r) := {/ e ,#?-oo I ll/IU;- < 00} 

a modulation space for the irreducible unitary representation tt : G — > £§(Jf?) with 
respect to the decomposition g e ~ glx g 2 e and the window vector <\> £ \ {0}. □ 

Example 5.5. For any choice of <f> £ Jrffoo \ {0} in Definition [53] we have 

Indeed, this equality holds since ||^/|U2(j e ) = ||^|| ■ ||/|| for every f £ Jf? (see 
Proposition 14.101 above) . □ 



5.3. A specific irreducible representation on L 2 (&). We are going to con- 
struct here some irreducible representations on the Hilbert spaces of the form L 2 (ff), 
where G can be any coadjoint orbit of a nilpotent Lie group. A different construc- 
tion involving the Moyal product (see Definition 14. 1\ was used in Def. 2.19 in the 
paper BB09c in order to get a representation 7r^ with the same representation 
space L 2 (&). The modulation spaces for 7r# turned out to be relevant for estab- 
lishing the continuity properties of the pseudo-differential operators obtained by the 
Weyl-Pedersen calculus for any unitary representation associated with the coadjoint 
orbit 0. (See also Remark l4~T3l ) 
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Proposition 5.6. Let Z be the center of the connected, simply connected, nilpotent 
Lie group G with the corresponding Lie algebra 3 Cg. Endow the coadjoint orbit € 
with a Liouville measure and define 

n:GK AdS ^%(L 2 (0)), (SrfoY)/)® = e^/CAd^" 1 ^)- 
Then the following assertions hold: 

(1) The group G := G KAd is nilpotent and its center is Z x 3. 

(2) 7r is a unitary irreducible representation of G. 

(3) Let us denote by g — g ix a( j Qq the Lie algebra of G (where Qq stands for the 
abelian Lie algebra with the same underlying linear space as q) and define 

~ ( (0,X 3 ) forj = l,...,n, 
3 \(X,_„,0) forj=n + l,...,2n. 

Then X\ . . . . , X 2n is a Jordan-Holder basis in g and the corresponding pre- 
dual for the coadjoint orbit & C g* associated with the representation 7r 
is 

9e = 0e X fle C Q, 

where e is the set of jump indices for '. 

(4) The space of smooth vectors for the representation n is ^{ff). 

Proof. ([I]) Recall that the multiplication in the semi-direct product group G is given 

by 

(Si, Yi) • (02, Y 2 ) = (0i0 2 ,Y 1 + Ad G (0i)Y 2 ) 
while the bracket in the corresponding Lie algebra g = g x a d is defined by 

[(X 1 ,Y 1 ),(X 2 ,Y 2 )} = ([X 1 ,X 2 ],[X 1 ,Y 2 }-[X 2 ,Y 1 ]). (5.3) 

An inspection of these equations quickly leads to the conclusion that g is a nilpotent 
Lie algebra with the center 3x3. 

© If (01, Yi), (02, Y 2 ) G G and / G L 2 {ff), then for £ G 6 we have 

n( 9l , Y 1 )(n(g 2 , Y 2 )/)(£) = e^*) (n(g 2 , Y 2 )f)(Ad* G (g^)0 

= e i ^^)e i < Ad =^ 1 )f^>/(Ad^(0 2 - 1 )Ad^(0 1 - 1 )O 

= e i ^^ +Ad °^)^>/(Ad£((0 1 2 )- 1 )O 

hence 7?(0i, Yi)7r(0 2 , Y 2 ) — tt((0i, Y)(02, Next note that the representation 7r 
is unitary since the Liouville measure on 6 is invariant under the coadjoint action 
of G. 

To see that tt is irreducible, let T: L 2 (&) — > L 2 (&) be any bounded linear 
operator satisfying T7r(<7,Y) = 7r(0,Y)T for arbitrary (,0,Y) G G. We have to 
check that T is a scalar multiple of the identity operator on L 2 (&). By applying 
the assumption for g = 1 G G we see that T belongs to the commutant of the family 
of multiplication operators by the functions in the family {e^ y ' ' | Y G g} C L°°(<??). 
On the other hand, we recall that the mapping 

ff^R d , ^((tX n ),...,(tX ]d )) 

is a global chart which takes the Liouville measure of 6 to a Lebesgue measure 
on R d (see for Remark |4.3|) . Then we can use the Fourier transform to see that the 
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linear subspace generated by {e 1 ^ Y G g} is weak*-dense in L°°(G) (~ L 1 ^)*). 
Therefore the operator T: L 2 (G) — > L 2 (G) commutes with all the multiplication 
operators by functions in L°°(G), and then it has to be in turn the multiplication 
operator by some function <f> G L°°(G). Now, by using the assumption that T 
commutes with n(g, 0) for arbitrary g G G, it easily follows that <p has to be a 
constant function since the coadjoint action of G on the orbit G is transitive. 

It is straightforward to check that Xi , . . . , Xi n is a Jordan- Holder basis 
in g. Next note that S^{G) is contained in the space of smooth vectors for the 
representation tt and for arbitrary / G 5^[G) and (X, Y) eg we have 

(V£g^) (d5r(X,y)/)(0=i«,lO/(0 + 4 /(C°e tad ^). (5.4) 

di t=o 

It then follows by a straightforward application of Prop. 2.4.1 in |Pe84j and by 
Lemmas 1.4.1 and 1.5.1 in [Pc89] that the set of jump indices for the coadjoint 
orbit G \se = {j 1 ,...,j d ,n + ji,...,n + jd}, and then g-g = g e x g e C g. 

([!]) It follows by ()5.4p and by Lemmas 1.4.1 and 1.5.1 in |Pe89j again that there 
exists a polynomial chart on G such that in the corresponding chart, the associative 
algebra generated by the image of d7r contains all the linear partial differential op- 
erators with polynomial coefficients. This implies that the space of smooth vectors 
for the representation tt is equal to ,5^{G), as asserted. □ 

Corollary 5.7. Assume the setting of Provosition [5751 The ambiguity function 

rf: L 2 {G) x L 2 {G) - L 2 fa) = L 2 (g e x 0e ) 

for the representation tt: G — > M(L (Jf?)) is given by the formula 

-i<£ : /c sad s x Fds) 



(^f)(X,Y)=Je —o 7(£)<K£°e ad «*)d£ 

for arbitrary X, Y G g e and f,<fi G L 2 (G). 

Proof. For every f,h€ L 2 (G) we have 

K^/)(X, F) = (/ | i(exp G (X, y))0)^ (<T) . (5.5) 

On the other hand, for the element (X, Y) G g we have 

l l 
exp d (X,Y) = (exp G X, J Ad G (exp G (sX))y ds) = (exp G X, J e sad » x Yds) 
o o 

(see Example 15. 3p hence 

1 d x 

^(exp G (X,y))^)(0=e 1<5, o° 9 ydS V(Ad G (exp G (-*))£) 

i(?,/c sad £> x Yd s ) , „ 

= e o ^oe^'^) 
and then the conclusion follows by (|5.5[) . □ 
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Remark 5.8. Assume the setting of the above Proposition 15.61 It follows by 
Corollary 15.71 along with Schur's criterion for integral operators that there ex- 
ists a constant Cj> > such that for every F S L 2 (ff) and Y e g e we have 
\\^F{;Y)\\ L 2 {ae) < Cs\\F\\ L . {e) , hence ||F|| M a.. (;f) < C*\\F\\ L * {e) . Therefore 

there exists a continuous inclusion map L 2 {0) ^ Ml'°°(n). See also |BB09bj for 
similar inclusion maps for the modulation spaces in the setting of the magnetic 
Weyl calculus on nilpotent Lie groups. □ 

Example 5.9. Assume that g is two-step nilpotent Lie algebra, that is, we have 
[flj [fljfl]] = {0}- Let & C g* be any nontrivial coadjoint orbit and pick £ S &■ If 
we denote by 3 the center of g, then 

^={£60*1 a = Cola}, 
since ^ is a flat orbit. Then by Corollary 15.71 along with the fact that [g,g] C 3 we 
get 

1 

~ f -i(£. r c s ad s x y ds) 

K/)(x,y)=/ e '0 /(^(^^. x )d( 

= /e^ + ^^>/(0^ + e°ad fl X)de 

<? 

= e^"'™ | e- l <Mf@W+ £0 °ad fl X)d£. 

The above formula and suitable global coordinates on ^, one shows that the am- 
biguity function of the representation tt agrees with the ambiguity function of the 
Schrodinger representation of a certain Heiscnberg group, as defined in [GrOl . □ 

Acknowledgment. Partial financial support from the grant PNII - Programme 
"Idei" (code 1194) is acknowledged. 



References 



[An69] R.F.V. Anderson, The Weyl functional calculus. J. Functional Analysis 4(1969), 240- 
267. 

[An72] R.F.V. Anderson, The multiplicative Weyl functional calculus. J. Functional Analysis 
9(1972), 423-440. 

[BB09a] I. Beltita, D. Beltita, Magnetic pseudo-differential Weyl calculus on nilpotent Lie 
groups. Ann. Global Anal. Geom. 36 (2009), no. 3, 293-322. 

[BB09b] I. Beltita, D. Beltita, Uncertainty principles for magnetic structures on certain coad- 
joint orbits. J. Geom. Phys. (to appear). 

[BB09c] I. Beltita, D. Beltita, Modulation spaces of symbols for representations of nilpotent 
Lie groups. Preprint arXiv:0908.3917v2 [math.AP]. 

[BB09d] I. Beltita, D. Beltita, A survey on Weyl calculus for representations of nilpotent Lie 
groups. In: S.T. Ali, P. Kielanowski, A. Odzijewicz, M. Schlichenmeier, Th. Voronov 
(eds.), XXVIII Workshop on Geometric Methods in Physics, AIP Conf. Proc, Amer. 
Inst. Phys., Melville, NY (to appear). (See preprint larXiv:0910. 19941 /1 [math.AP].) 

[Br56] F. Bruhat, Sur les representations induites des groupes de Lie. Bull. Soc. Math. France 
84 (1956), 97-205. 

[Ca76] P. Cartier, Vecteurs differentiables dans les representations unitaires des groupes 
de Lie. In: Seminaire Bourbaki (1974/1975), Exp. No. 454. Lecture Notes in Math., 
Vol. 514, Springer, Berlin, 1976, pp. 20-34. 

[C068] I. COLOJOARA, Elements of Spectral Theory. (Romanian) Ed. Acad., Bucharest, 1968. 



SMOOTH VECTORS AND WEYL-PEDERSEN CALCULUS 



23 



[Co84] L. Corwin, Matrix coefficients of nilpotent Lie groups. In: Lie Group Representations, 
III (College Park, Md., 1982/1983). Lecture Notes in Math., 1077, Springer, Berlin, 
1984, pp. 1-11. 

[CG90] L.J. Corwin, F.P. Greenleaf, Representations of Nilpotent Lie Groups and Their 
Applications. Part I. Basic theory and examples. Cambridge Studies in Advanced Math- 
ematics, 18. Cambridge University Press, Cambridge, 1990. 

[CGP77] L.J. Corwin, F.P. Greenleaf, R. Penney, A general character formula for irreducible 
projections on L 2 of a nilmanifold. Math. Ann. 225 (1977), no. 1, 21-32. 

[CM96] L. Corwin, C.C. MOORE, L p matrix coefficients for nilpotent Lie groups. Rocky Moun- 
tain J. Math. 26 (1996), no. 2, 523-544. 

[DDJP09] E. Damek, J. Dziubanski, Ph. Jaming, S. Perez-Esteva, Distributions that arc 
convolvable with generalized Poisson kernel of solvable extensions of homogeneous Lie 
groups, Math. Scand. 105(2009), 31-65. 

[Dix74] J. Dixmier, Algebres Enveloppantes. Cahiers Scicntifiques, Fasc. XXXVII. Gauthicr- 
Villars Editeur, Paris-Brussels-Montreal, Que., 1974. 

[DM78] J. Dixmier, P. Malliavin, Factorisations dc fonctions et de vecteurs indefiniment 
differentiables. Bull. Sci. Math. (2) 102 (1978), no. 4, 307-330. 

[FG92] H.G. Feichtinger, K. GROCHENIG, Gabor wavelets and the Heisenberg group: Gabor 
expansions and short time Fourier transform from the group theoretical point of view. 
In: Wavelets, Wavelet Anal. Appl., 2, Academic Press, Boston, MA, 1992, pp. 359-397. 

[Fo89] G.B. Folland, Harmonic Analysis in Phase Space. Annals of Mathematics Studies, 
122. Princeton University Press, Princeton, NJ, 1989. 

[Ga47] L. Carding, Note on continuous representations of Lie groups. Proc. Nat. Acad. Sci. 
USA 33 (1947), 331-332. 

[Ga60] L. Garding, Vecteurs analytiqucs dans les representations des groups de Lie. Bull. Soc. 
Math. France 88 (1960), 73-93. 

[GK69] I.C. GOHBERG, M.G. Krein, Introduction to the Theory of Linear Nons elf adjoint Op- 
erators. Translations of Mathematical Monographs, Vol. 18, American Mathematical 
Society, Providence, RI, 1969. 

[GrOl] K. GROCHENIG, Foundations of Time- Frequency Analysis. Applied and Numerical Har- 
monic Analysis. Birkhauser Boston, Inc., Boston, MA, 2001. 

[GH99] K. GROCHENIG, C. Heil, Modulation spaces and pscudodifferential operators. Integral 
Equations Operator Theory 34 (1999), no. 4, 439-457. 

[GZ01] K. GROCHENIG, G. Zimmermann, Hardy's theorem and the short-time Fourier trans- 
form of Schwartz functions. J. London Math. Soc. (2) 63 (2001), no. 1, 205-214. 

[HN85] B. Helffer, J. NOURRIGAT, Hypoellipticite Maximale pour des Operateurs Polyndmes 
de Champs de Vecteurs. Progress in Mathematics, 58. Birkhauser Boston, Inc., Boston, 
MA, 1985. 

[Ho65] G. HOCHSCHILD, The Structure of Lie Groups. Holdcn-Day, Inc., San Francisco-London- 
Amsterdam, 1965. 

[Hor79] L. Hormander, The Weyl calculus of pscudodifferential operators. Comm. Pure Appl. 
Math. 32 (1979), no. 3, 360-444. 

[Hor07] L. Hormander, The Analysis of Linear Partial Differential Operators. III. Pseudo- 
differential operators. Reprint of the 1994 edition. Classics in Mathematics. Springer, 
Berlin, 2007. 

[How77] R.E. Howe, On a connection between nilpotent groups and oscillatory integrals asso- 
ciated to singularities. Pacific J. Math. 73 (1977), no. 2, 329-363. 

[How80] R. Howe, On the role of the Heisenberg group in harmonic analysis. Bull. Amer. Math. 
Soc. (N.S.) 3 (1980), no. 2, 821-843. 

[How84] R.E. Howe, A symbolic calculus for nilpotent groups. In: Operator Algebras and Group 
Representations (Neptun, 1980), vol. I, Monogr. Stud. Math., 17, Pitman, Boston, MA, 
1984, pp. 254-277. 

[HRW84] R. Howe, G. Ratcliff, N. Wildberger, Symbol mappings for certain nilpotent 
groups. In: Lie group representations III (College Park, Md., 1982/1983), Lecture 
Notes in Math., 1077, Springer, Berlin, 1984,pp. 288-320. 

[IMP07] V. Iftimie, M. Mantoiu, R. Purice, Magnetic pscudodifferential operators. Publ. Res. 
Inst. Math. Sci. 43 (2007), no. 3, 585-623. 



24 



INGRID BELTITA AND DANIEL BELTITA 



[IMP09] V. Iftimie, M. Mantoiu, R. PuriCE, Commutator criteria for magnetic pseudodiffer- 

ential operators. Preprint arXiv:0902.0513v2 [math-ph]. 
[Ki62] A. A. Kirillov, Unitary representations of nilpotent Lie groups. (Russian) Uspehi Mat. 

Nauk 17 (1962), no. 4 (106), 57-110. 
[Ki76] A. A. Kirillov, Elements of the Theory of Representations. Grundlehren der Mathc- 

matischen Wissenschaften, Band 220. Springer- Verlag, Berlin-New York, 1976. 
[Ki04] A. A. Kirillov, Lectures on the Orbit Method. Graduate Studies in Mathematics, 64. 

American Mathematical Society, Providence, RI, 2004. 
[Mai07] J.-M. Maillard, Explicit star products on orbits of nilpotent Lie groups with square 

integrable representations. J. Math. Phys. 48 (2007), no. 7, 073504. 
[Man91] D. Manchon, Calcul symbolique sur les groupes de Lie nilpotents ct applications. J. 

Funct. Anal. 102 (1991), no. 1, 206-251. 
[Man95] D. Manchon, Operateurs pseudodifferentiels et representations unitaires des groupes 

de Lie. Bull. Soc. Math. France 123 (1995), no. 1, 117-138. 
[MP04] M. Mantoiu, R. Purice, The magnetic Weyl calculus. J. Math. Phys. 45 (2004), no. 4, 

1394-1417. 

[MP09] M. Mantoiu, R. Purice, The modulation mapping for magnetic symbols and opera- 
tors. Preprint arXiv:0907.5399V l [math.FA] 
[Me83] A. Melin, Parametrix constructions for right invariant differential operators on nilpo- 
tent groups. Ann. Global Anal. Geora. 1 (1983), no. 1, 79-130. 
[Mi82] K.G. Miller, Invariant pseudodifferential operators on two-step nilpotent Lie groups. 

Michigan Math. J. 29 (1982), no. 3, 315-328. 
[Mi86] K.G. Miller, Invariant pseudodifferential operators on two-step nilpotent Lie groups. 

II. Michigan Math. J. 33 (1986), no. 3, 395-401. 
[Pe84] N.V. Pedersen, On the infinitesimal kernel of irreducible representations of nilpotent 

Lie groups. Bull. Soc. Math. France 112 (1984), no. 4, 423-467. 
[Pe88] N.V. Pedersen, On the symplectic structure of coadjoint orbits of (solvable) Lie groups 

and applications. I. Math. Ann. 281 (1988), no. 4, 633-669. 
[Pe89] N.V. Pedersen, Geometric quantization and the universal enveloping algebra of a 

nilpotent Lie group. Trans. Amer. Math. Soc. 315 (1989), no. 2, 511-563. 
[Pe94] N.V. Pedersen, Matrix coefficients and a Weyl correspondence for nilpotent Lie groups. 

Invent. Math. 118 (1994), no. 1, 1-36. 
[Pu67] L. Pukanszky, Lecons sur les Representations des Groupes. Monographies de la Societe 

Mathematique do France, No. 2, Dunod, Paris, 1967. 
[RT87] YA.V. Radyno, Yu.V. Titkov, Nuclear spaces of smooth vectors. (Russian) Dokl. 

Akad. Nauk BSSR 31 (1987), no. 8, 677-679 
[Ro76] F. Rouviere, Sur la resolubilite locale des operateurs bi-invariants. Ann. Scuola Norm. 

Sup. Pisa CI. Sci. (4) 3 (1976), no. 2, 231-244. 
[Sch66] H.H. SCHAEFER, Topological Vector Spaces. The Macmillan Co., New York; Collier- 

Macmillan Ltd., London, 1966. 
[Tr67] F. Treves, Topological Vector Spaces, Distributions and Kernels. Academic Press, New 

York-London, 1967. 

[Wa72] G. Warner, Harmonic Analysis on Semi-Simple Lie Groups. I. Die Grundlehren der 
mathematischen Wissenschaften, Band 188. Springer- Verlag, New York-Heidelberg, 
1972. 

[We28] H. Weyl, Gruppentheorie und Quantenmechanik. Verlag S. Hirzel, Leipzig, 1928. 



Institute of Mathematics "Simion Stoilow" of the Romanian Academy, P.O. Box 1- 
764, Bucharest, Romania 

E-mail address: Ingrld.BeltitaOimar.ro 

E-mail address: Daniel.BeltitaOimar.ro 



